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Abstract. This note concerns a search for publications in which the 
pragmatic concept of a test as conducted in the practice of software 
testing is formalized, a theory about software testing based on such a 
formalization is presented or it is demonstrated on the basis of such a 
theory that there are solid grounds to test software in cases where in 
principle other forms of analysis could be used. This note reports on the 
way in which the search has been carried out and the main outcomes of 
the search. The message of the note is that the fundamentals of software 
testing are not yet complete in some respects. 



1 Introduction 

In practice, it is virtually unavoidable that software has faults. This implies that 
there are normally risks involved in using software. In order to reduce the risks 
as much as possible, developed software is put to tests before it is released. Soft- 
ware testing definitely tends to form an important part of software development. 
A test as conducted in the practice of software testing is an activity in which 
certain code is executed under certain conditions, certain effects of the execu- 
tion are observed, and an evaluation of the observed effects is made. A sound 
understanding of this pragmatic concept is a prerequisite for the advancement 
of software testing. 

To check whether more or less appropriate formalized versions of the prag- 
matic concept of a test, as well as theories based on them, already exist, a search 
for witnessing publications has been carried out. An issue of utmost importance 
is the issue of whether or not there are solid grounds to test software in cases 
where in principle other forms of analysis could be used. For that reason, the 
above-mentioned search has been combined with a search for publications treat- 
ing this issue. This note reports on the way in which the search has been carried 
out and the main outcomes of the search. In the concluding remarks, the main 
outcomes of the search are roughly summarized in the form of two remarkable 
observations, which clearly reflect the fact that the fundamentals of software 
testing are not yet complete in some respects. Therefore, it may well be that 
these observations will serve as triggers of future work on the fundamentals of 
software testing. 



2 The Approach to Carry out the Search 



For the search, Google Scholar and the search engines of relevant journal col- 
lections and bibliographies are available. The search engines of relevant journal 
collections and bibliographies have the option to search for publications in which 
given terms occur in their title, abstract or keywords, but none cover all relevant 
literature. Google Scholar does not have this option, i.e. only full text search 
is possible, but it covers all literature covered by the search engines of relevant 
journal collections. 

In a preparatory phase of the search, the following specifics were found: 

— the term "software testing" rarely occurs in publications whose subject is 
not software testing; 

— in the early days of software testing, the term "program testing" is often 
used as a synonym for "software testing" . 

This led to the following more or less systematic way to carry out the search: 

— first, divide the past from 1951 in periods of five years and search for relevant 
publications in each period seperately by means of Google Scholar; 

— next, check for each relevant publication found in the previous step whether 
it references additional relevant publications; 

— then, repeat the previous step until no additional relevant publications are 
found; 

— finally, search for relevant publications by means of the search engines of 
relevant journal collections and bibliographies. 

The number of publications found by a full text search with Google Scholar 
using the search term "program testing" OR "software testing" is about 
15,000. Google Scholar shows only the 1,000 best matching publications, but by 
searching in periods of five years I could go through the 6,300 best matching 
publications. The last step is meant as a double-check: it should yield very few 
new relevant publications. The following journal collections and bibliographies 
are considered relevant: ACM Digital Library, SpringerLink, ScienceDirect, IEEE 
Xplorc, ACM Guide, DBLP Computer Science Bibliography, and the Collection 
of Computer Science Bibliographies. 

The publications that were considered relevant to the purpose of the search 
in the first step turned out to be strongly connected by citations. It proved to 
be unnecessary to do the second step more than twice. 

In a previous search for publications, which is reported on in [2l] . the term 
for the concept concerned was a term that occurs in extremely many publications 
virtually unrelated to the concept. Consequently, that search had to be carried 
out in a more sophisticated way. 

3 The Main Outcomes of the Search 

Recall that the publications searched for are publications in which the pragmatic 
concept of a test as conducted in the practice of software testing is formalized, a 
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theory about software testing based on such a formalization is presented or it is 
demonstrated on the basis of such a theory that there are solid grounds to test 
software in cases where in principle other forms of analysis could be used. 

Bear in mind that a test as conducted in the practice of software testing is 
an activity in which certain code is executed under certain conditions, certain 
effects of the execution are observed, and an evaluation of the observed effects 
is made. It turns out that there is no electronically available publication on 
software testing in which this pragmatic concept of a test is formalized. 

Up to around 1970, the term "program testing" is used instead of the term 
"software testing". The oldest electronically available publication in which the 
former term is used seems to be an article by Perry, namely [25], which is 
published in 1952. To our knowledge, the oldest electronically available arti- 
cle on software testing isan article by Sauder, namely |28j . which is published 
in 1962. Relatively in-depth theoretical investigations into various important 
aspects of software testing begin around 1975. The aspects that are investi- 
gated include test selection (see e.g. |3ll4l6ll7l 26 27 38 24 9 ), test evaluation 
(see e.g. [5I35I36I22I31I7I8I20] ). testability (see e.g. [34118132140] ). testing com- 
plexity (see e.g. [30137123] ). and reliability /dependability of tested software (see 
e.g. I4I33I13I21)] ). The theories about software testing that came into being con- 
solidate the results of those investigations. No trace of the pragmatic concept of 
a test outlined above is found in those theories. In other words, they do not pro- 
vide a sound understanding of the concept of a test as conducted in the practice 
of software testing. 

Among the theories in question, the theories presented in |11|15|16|10|39|12|2|1] 
are usually considered the most interesting ones. Programs are identified in those 
theories with objects of various kinds, including programs in some programming 
language, flowcharts, total functions, structures in the sense of mathematical 
logic, and even objects of an arbitrary kind related in specific ways with objects 
of two other arbitrary kinds — with which specifications and tests are identified. 
With the exception of programs in some programming language, these (semi-) 
formalizations of the pragmatic concept of a program impede the introduction 
of suitable formalizations of the pragmatic concept of a test outlined above. 

Several existing theories about software testing are introduced by means of a 
brief outline of the pragmatic concept of a test that looks much like the outline 
given above, but surprisingly even in those theories no trace of the concept is 
found. It looks as if, at the very outset of the development of existing theories 
about software testing, it is immediately forgotten that in software testing a test 
involves the execution of code and observation of effects of the execution. This 
is amazing because what is forgotten is exactly what sets software testing apart 
from other forms of software analysis. Recommendation Z.500 of the Interna- 
tional Telecommunication Union QI5] is one of the few publications that goes 
more deeply into what is involved in a test, but this standardization document 
does not provide a formalization of the concept. 

It turns out that there is no electronically available publication on software 
testing that seriously treats the issue of whether or not there are solid grounds 
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to test software in cases where in principle other forms of analysis could be used. 
If the rationale of software testing is given, it is invariably that software testing 
is less expensive, less time-consuming and less error-prone than any other form 
of analysis that could be used. This is however a collection of claims for which 
not a shred of evidence is produced in the literature on software testing. This 
is amazing because it means that the rationale of software testing has not yet 
been settled after fifty years of actual software testing. Clearly, the lack of a 
theory about software testing that allows of setting software testing apart from 
other forms of software analysis has impeded a rationale with a sound theoretical 
basis. 

4 Miscellaneous 

In Section O the main outcomes of the search are outlined. That outline could 
raise the question what most publications on software testing are about. It turns 
out that they mainly concern the following: 

— theories and techniques pertaining to particular aspects of software testing, 
such as test selection, test evaluation, testability, testing complexity, relia- 
bility of tested software, compositional testing, and regression testing; 

— theories and techniques pertaining to software testing with regard to par- 
ticular kinds of behaviour, such as input-output behaviour and interactive 
behaviour; 

— issues related to the testing of software developed according to particular 
paradigms, such as object-oriented software, component-based software and 
service-oriented software; 

— designs of, analyses of, and experiences with specific software testing tools 
and techniques. 

It is striking that most of these publications give little insight into the pragmatic 
concept of a test as conducted in the practice of software testing. 

With regard to the journals in which theoretical articles on software testing 
are published, it is noticeable that from about 1975 a relatively large part of 
the theoretical articles on software testing is published in "IEEE Transactions 
on Software Engineering". It is visible that since 1991 the journal "Software 
Testing, Verification and Reliability" is becoming increasingly important. 

5 Concluding remarks 

To check whether more or less appropriate formalized versions of the pragmatic 
concept of a test, as well as theories based on them, already exist, a search for 
witnessing publications have been carried out. From the main outcomes of the 
search, it must be concluded that: 

— there does not exists a formalized version of the pragmatic concept of a test 
as conducted in the practice of software testing; 
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— there does not exist a theory about software testing that provides a sound 
understanding of the pragmatic concept of a test; 

— no attention is paid to giving a rationale of software testing that has a solid 
theoretical basis. 

Some remarkable observations are: 

— in the development of existing theories about software testing, what sets 
software testing apart from other forms of software analysis, namely that a 
test involves the execution of code and observation of effects of the execution, 
is completely forgotten; 

— there has not been any serious study of the issue of whether or not there 
are solid grounds to test software in cases where in principle other forms of 
analysis could be used. 
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